HDDS-4126. Freon key generator should support >2GB files. #3054
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jojochuang
approved these changes
Feb 8, 2022
Contributor
jojochuang
left a comment
jojochuang
left a comment
There was a problem hiding this comment.
+1 LGTM
This is trivial so no test is required.
adoroszlai
approved these changes
Feb 8, 2022
Contributor
adoroszlai
left a comment
adoroszlai
left a comment
There was a problem hiding this comment.
Thanks @ArafatCloudera for the improvement.
Tested:
$ ozone sh volume create vol1
$ ozone sh bucket create --layout FILE_SYSTEM_OPTIMIZED vol1/fso
$ ozone freon dfsg -n1 -t1 --path ofs://om/vol1/fso -p test -s 4294967296
...
$ ozone fs -ls ofs://om/vol1/fso/test
Found 1 items
-rw-rw-rw- 3 hadoop hadoop 4294967296 2022-02-08 13:34 ofs://om/vol1/fso/test/0
adoroszlai
reviewed
Feb 8, 2022
Contributor
adoroszlai
left a comment
adoroszlai
left a comment
There was a problem hiding this comment.
@ArafatCloudera I have just noticed that there are a few other cases where the same limit applies:
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/StreamingGenerator.java
66: private int fileSize;
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/S3KeyGenerator.java
73: private int fileSize;
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/HadoopDirTreeGenerator.java
73: private int fileSizeInBytes;
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/containergenerator/BaseGenerator.java
44: private int keySize;
Would you mind changing these as well to long?
Contributor
Author
Sure will get it done !! |
Contributor
Author
|
@adoroszlai I could only make changes for StreamingGenerator.java & HadoopDirTreeGenerator.java I could not make changes to S3KeyGenerator.java because the RandomStringUtils.randomAscii(fileSize) function on line:122 takes only Integer as an argument. I could not make changes to BaseGenerator.java because a lot of the code would have to be rewritten in other files. |
adoroszlai
reviewed
Feb 9, 2022
| while (writtenBytes < getKeySize()) { | ||
| int currentChunkSize = | ||
| Math.min(getKeySize() - writtenBytes, chunkSize); | ||
| Math.min(getKeySize() - writtenBytes, chunkSize); |
Contributor
There was a problem hiding this comment.
GeneratorDatanode only has whitespace change. Can you please omit it from the patch?
Contributor
That's fine. Thanks for checking. |
Contributor
|
Thanks @ArafatCloudera for the patch, @jojochuang for the review. |
errose28
added a commit
to errose28/ozone
that referenced
this pull request
Mar 30, 2022
* HDDS-4944: (268 commits) HDDS-6366. [Multi-Tenant] Disallow specifying custom accessId in OzoneManager (apache#3166) HDDS-6275. [Multi-Tenant] Add feature documentation and CLI quick start guide (apache#3095) HDDS-6063. [Multi-Tenant] Use VOLUME_LOCK in read and write requests, and some minor refactoring (apache#3051) HDDS-6214. [Multi-Tenant] Fix KMS Encryption/Decryption (apache#3010) HDDS-6322. Fix Recon getting inccorrect sequenceNumber from OM (apache#3090) HDDS-5913. Avoid integer overflow when setting dfs.container.ratis.lo… (apache#2785) HDDS-6313. Remove replicas in ContainerStateMap when a container is deleted (apache#3086) HDDS-6186. Selective checks: skip integration check for unit test changes (apache#3061) HDDS-6310. Update json-smart to 2.4.7. (apache#3080) HDDS-6190. Cleanup unnecessary datanode id path checks. (apache#2993) HDDS-6304. Add enforcer to make sure ozone.version equals project.version (apache#3075) HDDS-6309. Update ozone-runner version to 20220212-1 (apache#3079) HDDS-6293. Allow using custom ozone-runner image (apache#3072) HDDS-4126. Freon key generator should support >2GB files. (apache#3054) HDDS-6088. Implement O3FS/OFS getFileChecksum() using file checksum helpers - addendum: fix checkstyle HDDS-6088. Implement O3FS/OFS getFileChecksum() using file checksum helpers. (apache#2935) HDDS-6084. [Multi-Tenant] Handle upgrades to version supporting S3 multi-tenancy (apache#3018) HDDS-6257. Wrong stack trace for S3 errors (apache#3066) HDDS-6278 Improve memory profile for listStatus API call. (apache#3053) HDDS-6285. ozonesecure-mr intermittently failing with timeout downloading packages (apache#3057) ...
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
fileSize parameter in HadoopFsGenerator is an int which limits it to max 2GB. But works well with long.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-4126
How was this patch tested?
Na